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The Centers for Disease Control and Prevention’s Sustainable Management 
Development Program (SMDP) partners with low- and middle-resource 
countries to develop management capacity so that effective global public 
health programs can be implemented and better health outcomes can be 
achieved. The program’s impact however, was variable. Hence, there was a 
need to both engage in a strategic planning process and collect useful data to 
inform the process. We therefore designed a qualitative evaluation and 
findings that emerged concerning our program’s contribution to individual 
career advancement and professional growth; the need for institutional 
support and a champion to move public health management capacity 
development efforts forward in low- and middle-resource countries; and 
interest in diverse professional learning opportunities contributed to program 
improvement and suggested new strategic directions for CDC’s global public 
health management service delivery. Our inquiry provides a concrete 
example of how qualitative methods, specifically key informant interviews, can 
provide useful data for strategic planning within public health settings. It may 
be useful to readers who are interested in conducting strategic planning within 
public health and other related areas including health care, mental and 
behavioral health, and the social sciences. Keywords: Key Informant 
Interviews; Qualitative Evaluation; Strategic Planning; Global Health; Public 
Health Management; Program Evaluation 


Introduction 


Our program 

The Sustainable Management Development Program (SMDP) is a global public health 
management capacity development program at the Centers for Disease Control and 
Prevention (CDC). It partners with Ministries of Health, educational institutions, and non¬ 
governmental organizations (NGOs) in low- and middle-resource countries to develop global 
public health management capacity so that effective public health programs can be designed 
and implemented, and better health outcomes can be achieved. 

CDC’s approach to developing global public health management capacity involved 
sponsoring the 6-week long Management for International Public Health (MIPH) course in 
Atlanta, GA. Participants came from foreign Ministries of Health, NGOs, and academic 
institutions in low- and middle-resource countries. CDC trained 379 individuals from 66 
countries from 1992 through 2008 (Figure 1) in the course. CDC staff provided follow-up 
technical assistance (either remotely from Atlanta or through in person site visits) when 
funding was available to support the expectation that alumni would subsequently develop 
public health management training and capacity development programs in their countries of 
origin. 

The impact of our program’s approach was variable. In some countries, our course 
alumni implemented public health management programs that improved the perfonnance of 
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public health teams, initiatives, and organizations (Sogoric et al., 2009; Umble et al., 2009). 
In other countries, our alumni did not establish any public health management capacity 
development programs at all, or did not successfully maintain the programs they did establish, 
thus providing little or no evidence of our program’s impact. Consequently, we decided to 
evaluate how our program could be improved and simultaneously contribute useful data to a 
broader strategic planning process that our program was undertaking. 

Figure 1: Map of countries of origin of all MIPH course alumni from 1992-2008 (n=379) 



Literature 

Qualitative evaluation and methods in strategic planning. 

We decided to conduct a qualitative evaluation because, according to Patton: “The 
purpose of interviewing ... is to allow us to enter into the other person’s perspective” (Patton, 
2002, p. 341). This is what we wanted to accomplish. We wanted to allow our alumni in 
low- and middle-resource countries to give voice to their experiences in order to improve our 
program and inform our strategic planning process. We thus sought to learn from our alumni 
and better understand their challenges and successes in implementing public health 
management capacity programs in their countries of origin. We also wanted them to share 
their perspectives on how we could improve our program and better meet their needs. 

We subsequently conducted a literature review to help us shape the focus of our 
qualitative evaluation and our program’s larger strategic planning process. We found several 
examples in the literature regarding how qualitative methods can be used in evaluation and 
strategic planning. Vorhees (2008) provides a general overview of using interviews, focus 
groups, and group strategy sessions. 

Several articles describe using key informant interviews in strategic planning. 
Szydlowski and Smith (2009) use key informant interviews with chief information officers to 
better understand the implementation of information technology within hospital settings. Inui 
et al. (2009) detail using key informant interviews in evaluating a HIV care program in Kenya 
and for related strategic planning purposes. Gruffudd (2008) describes using key informant 
interviews in evaluating an air ambulance program and how these data contributed to strategic 
planning. Teng, Milton, and Mackenzie (2007) portray using key informant interviews in 
better understanding priority-setting within a provincial health services authority’s strategic 
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planning process. Lee (2006) interviews nurse managers of medical center inpatient units (i.e. 
key infonnants) to better understand how they adopt personal digital assistant technology, and 
describes how these results can be used to strategically plan for improving the implementation 
of technology. Begun and Kaissi (2005) illustrate using key informant interviews in 
ascertaining how hospitals conduct strategic planning. 

Additional articles within the literature demonstrate how other qualitative methods can 
be used within strategic planning. Farquhar, Parker, Schultz, and Israel (2006) describe using 
qualitative data from interviews, focus groups, and windshield tours in community health 
strategic planning. Takhar and Tipping (2008) detail using focus groups to provide qualitative 
data for a continuing medical education program’s strategic planning process. Madden, 
Martin, Downey, and Singer, (2005) and Martin, Shulman, Santiago-Sorrell, and Singer 
(2003) portray using document review, key informant interviews, and observations of group 
deliberations in developing case studies on hospital strategic planning. Levine, Plume, and 
Nelson (1997) use patient surveys that included a qualitative component to develop a case 
study to inform a hospital’s strategic planning efforts. 

In reviewing the results of our literature review, we decided specifically upon key 
infonnant interviews as our qualitative method for two reasons. The literature offered us 
specific guidance on using this method (Begun & Kaissi, 2005; Farquhar et ah, 2006; 
Gruffudd, 2008; Inui et ah, 2009; Lee, 2006; Madden et ah, 2005; Martin et ah, 2003; 
Szydlowski & Smith, 2009; Teng et ah, 2007;). We also wanted to leam as much as possible 
from our alumni’s lived experience in respect to developing public health management 
capacity within their countries, and we wanted to enter into their perspective (Patton, 2002). 

Literature gap. 

In conducting our literature review, we found an emerging evidence base on using 
qualitative methods in strategic planning, particularly within healthcare (i.e., hospitals, 
clinical practice) settings (Begun & Kaissi, 2005; Gruffudd, 2008; Lee, 2006; Levine et ah, 
1997; Madden et ah, 2005; Martin et ah, 2003; Szydlowski & Smith, 2009; Takhar & 
Tipping, 2008). We found fewer articles (Farquhar et ah, 2006; Inui et ah, 2009; Teng et ah, 
2007) providing concrete examples of using qualitative methods in strategic planning within 
public health settings. 

Problem, objectives and the benefit of our inquiry. 

In this article, we report on the qualitative evaluation we conducted by interviewing 
CDC’s public health management development course alumni in low- and middle-resource 
countries. In some countries, our course alumni had successfully implemented public health 
management capacity development programs (Sogoric et ah, 2009; Umble et ah, 2009). In 
other countries, our alumni did not establish any public health management capacity 
development programs at all, or did not successfully maintain the programs they did establish, 
thus providing little or no evidence of our program’s impact. Consequently, we wanted to 
address the problem of our program’s variable impact. 

Our objectives were twofold: (a) to contribute to the improvement of our public health 
management development program and (b) to produce useful data for strategic planning to 
position our program for additional impact. 

Our qualitative evaluation was designed to answer the following: 
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• What impact did CDC’s public health management development course 
(i.e., MIPH course) have on alumni and their organizations in middle- 
and low-resource countries? 

• What challenges did course alumni experience in establishing or 
maintaining public health management capacity development programs 
in their countries? 

• What did course alumni think of CDC potentially using new strategies 
to assist them in developing public health management capacity in their 
countries (e.g., leadership training, distance learning, and alternative 
fonns of technical assistance)? 

While our inquiry focused on improving our program and contributing data to our 
program’s larger strategic planning process, we hope that it will contribute to the emerging 
evidence-based literature on using qualitative methods in strategic planning and assist in 
filling the gap regarding using qualitative methods in strategic planning in public health. 

In tenns of the context that we brought to our qualitative inquiry, KEK, AK and EH 
were CDC employees working in the SMDP program. We intended to improve our program 
and provide useful data for strategic planning purposes. KEK worked in the capacity of 
SMDP’s evaluator, and co-led the evaluation with AK who worked as one of the program’s 
public health advisors. EH was SMDP’s chief, and additionally led its strategic planning 
process. GM was an evaluator in another organizational unit of CDC’s Center for Global 
Health who brought additional expertise in qualitative methods to the evaluation. She did not 
work directly in the SMDP program, although desired to contribute towards its improvement. 

Methods 


Qualitative evaluation design 

We decided to conduct a qualitative evaluation because we wanted to gain insight and 
context from our program participants and allow opportunity for their in-depth input 
(Farquhar et ah, 2006; Patton, 2002, 2008; Vorhees, 2008). Many of the issues that we 
wanted to address could not have been anticipated, and therefore could not have been 
incorporated into a highly structured method of data collection. 

We used Patton’s work to help shape our design (Patton, 2002, 2008). We found his 
Utilization-Focused Evaluation approach to be most helpful in relating back to our evaluation 
objectives because he describes program improvement as being a primary use of evaluation 
findings (Patton, 2008). SMDP staff members and its chief were the primary intended users 
of our evaluation findings. CDC global health leaders were additional stakeholders as they 
would ultimately be users of the program’s new strategic plan, and they had an interest in 
assuring that our program was strategically aligned for maximum public health impact. The 
needs of our stakeholders assisted us in focusing the evaluation through a series of meetings 
involving SMDP staff and an expert panel process engaging global health leaders from across 
CDC’s centers, institutes, and offices. 

Merriam’s (2009) description of descriptive qualitative research as an inquiry that is 
basic, naturalistic, and discovery-oriented further informed our design. Merriam describes this 
type of inquiry as staying closer to the data and to the surface of words and events than 
explanatory studies, and that it consists of “eclectic design” consisting of sampling strategies 
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(e.g., purposeful), data collection (e.g., open-ended interviews), and data analysis (e.g., 
categorization; Merriam, 2009). 

Sampling and recruitment 

We used purposeful sampling, which entails selecting infonnation-rich cases for 
study in depth... to permit inquiry into and understanding of a phenomenon ... [with the goal 
of] yielding insights” (Patton, 2002, p. 230). We divided countries into those with a program 
to develop public health management capacity (Group A) and those that either never had such 
program, or once had a program but it had not been successfully maintained (Group B). Our 
purpose, however, was neither to compare the two sets of countries nor to detennine factors 
that contributed to developing a successful program. Rather, we intended to learn more about 
the full spectrum of our MIPH course alumni’s experiences in developing, or trying to 
develop, public health management capacity development programs in their countries. We 
therefore selected countries that represented cases along a spectrum of program development 
from “never initiated” to “developed and maintained.” 

After we identified countries (Table 1), we ascertained who the most knowledgeable 
MIPH course alumni were by consulting with the SMDP staff members directly working 
with these countries. We wanted to ensure that we were speaking with key infonnants who 
were most expert about the public health management capacity development efforts that had 
been undertaken there. We found 17 key informants through this process and invited them by 
e-mail to participate in 1-hour telephone interviews. We followed up by e-mail at least once if 
they did not respond to our initial invitation. Of the 17 course alumni we invited to be 
interviewed, 14 participated. 

Instrument Design 

We developed an interview guide through an iterative process. KEK and AK 
developed the initial draft. EH and GM subsequently responded with feedback and comments 
through track changes. We did this until we considered the interview guide ready for use (i.e., 
when there were no remaining suggested changes or comments). Our interview guide 
ultimately comprised seven open-ended questions that were designed to elicit meaningful data 
for program improvement and strategic planning (the two objectives of our inquiry). The 
guide enabled us to approach our semi-structured interviews in a systematic way, while 
simultaneously allowing us to probe and explore within certain predetermined areas (Patton, 
2002). As such, it provided a framework within which we sequenced our questions, and made 
decisions about where to probe to understand in better detail our alumni’s experiences and 
perspectives (Patton, 2002). 

Our seven questions enabled us to inquire about our key informant’s thoughts on the 
key public health management challenges in their current position, whether they encountered 
any challenges in developing a public health management capacity development program in 
their countries, and whether (and how) participating in our program impacted their 
organization and them professionally. We further asked how our program could have better 
supported them in developing public health management capacity programs and explored 
their perspectives on what new services we could offer that would be most useful in building 
public health management capacity at a national level in their countries. Our closing question 
asked our key infonnants if there was anything else that we should have asked them, or if 
there was anything else that they wanted to share, in the hopes that we might distill 
additionally rich data and that our alumni would have the opportunity to express everything 
that they had wanted to share (Patton, 2002). We piloted the questions during our first two 
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telephone interviews and took notes on any questions (or components of questions) that 
seemed unclear. We subsequently amended questions because they were unclear to our key 
infonnants, even when we probed (Merriam, 2009). Hence, they did not meet the criteria for 
appropriate qualitative interview questions (Patton, 2002). 

Interview procedures 

Two of us (the first and second authors: KEK and AK) conducted the semi-structured 
interviews by telephone. All interviews were conducted in English, although English is not 
the first language of most of our key infonnants. One participant requested to respond to our 
interview questions in writing (rather than by telephone) because he believed himself to be 
more proficient in written English. We accommodated this request. We scheduled an hour for 
each telephone interview, and made plans to call our key informants either in their offices or 
homes, or via mobile phone, as per their indicated preferences. Depending upon the time of 
day in our participants’ countries, KEK and AK scheduled some calls outside of their nonnal 
working hours in order to accommodate global time differences. 

IRB approval was not required for our inquiry given that the infonnation was to be 
used for program improvement and was considered program evaluation (i.e. non-research). 
At the beginning of each interview, we verbally shared a statement of purpose with our key 
infonnants that reflected that we were conducting the interviews to assist our program in 
improving to have maximum impact in the future. We further shared that: (a) their country 
might be identified in a publication but that no reference would be made to them individually; 
(b) we did not foresee any risks related to participating; (c) participating would contribute to 
improving our course and informing our program’s strategic planning process; and (d) they 
could choose not to participate. Once our key infonnants verbally agreed, we proceeded with 
our interviews. This ensured that the key informants were aware of our evaluation’s purpose 
and that they understood that their participation was entirely voluntary. We went through all 
of our seven open-ended questions with each of our alumni. We concluded our interviews by 
thanking our key informants and encouraging them to contact us by email if they had any 
additional thoughts to share or if they had any questions after the interview. 

KEK was the primary interviewer, and AK was the primary note-taker capturing actual 
quotations from our key informants (Patton, 2002). While AK recorded the interviews 
verbatim, KEK additionally took notes during the interviews on key points that alumni shared 
(Patton, 2002). We maintained the same roles for all but two interviews during which AK 
perfonned both roles. KEK altered the order of questions on the basis of how each interview 
flowed, and we both probed responses as appropriate. AK prepared a draft transcript from her 
notes immediately after each interview. KEK then reviewed the transcript, and we 
immediately discussed and resolved any inconsistencies in our recollections of the interview. 
The text within our phone interview transcripts (capturing the actual words of key infonnants) 
became our data sources (Patton, 2002). The above process reflects how we prevented bias 
from occurring. 

Analysis procedures 

After completing all 14 interviews, KEK and AK each independently read line-by-line 
through one randomly selected transcript and attributed codes to segments of text, reflecting 
our initial attempt to organize the data (Patton, 2002). Our analysis was inductive as we hoped 
to discover patterns, themes and categories within our data. (Merriam, 2009; Patton, 2002). 
We subsequently created a draft codebook based on the approach detailed by MacQueen et al. 
(2007). 
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We then coded a second transcript together and revised the codebook. We separately 
coded the remaining interviews and reviewed them together, resolving disagreements when 
they occurred. Our process was one of cutting and sorting (i.e., identifying quotes that seem 
important and then arranging them into “piles” of things that go together (Ryan & Bernard, 
2003). Our initial intent was to analyze the data by question (i.e., to use structural coding). 
However, our data did not naturally come together in this way. Consequently, we used 
emergent rather than structural coding because the content of our data (rather than the 
structure of our questions) drove the development of our codes. After initially coding the 
transcripts, KEK and AK then jointly reviewed the coded paragraphs in each transcript noting 
important passages of the interviews in the margins. KEK and AK subsequently identified the 
topics repeating across interviews as themes (Merriam, 2009; Patton, 2002; Ryan & Bernard, 
2003). KEK and AK labeled them (beginning with the letter A), and recorded the number of 
key infonnants contributing to each theme. KEK and AK then corroborated, refuted and/or 
combined these initial themes, and nine themes pertinent to the goals of the inquiry 
emerged. The third and fourth authors then independently (LH and GM respectively) 
provided input on the coding and thematic results, resulting in the convergence of the initial 
nine themes into three major ones relevant for both program improvement and programmatic 
strategic planning (Merriam, 2009; Patton, 2002; Ryan & Bernard, 2003). All four authors 
engaged in interpreting the results of the analysis to ensure accuracy and limit potential 
biases. 


Results 


Participants 

Our 14 key informants participated in the MIPH course between 1993 and 2006. The 
average had participated approximately 7 years (mean = 6.71 years) before the time of our 
interviews. Eight of our participants were men (57.1%), and 6 were women (42.9%). They 
were employed by Ministries of Health, universities, NGOs, or CDC offices in their countries 
of origin at the time of our interviews. 

Table 1: Countries of origin of our key informants and program development status 
(n=14) 


Category A: 

Countries with public health 
management capacity development 
programs 

Category B: 

Countries that never had a public health 
management capacity developmen t program, 
or once had a program but it was not 
successfully maintained 

Botswana 

India 

Croatia 

Indonesia 

Ethiopia 

Israel 

Malawi 

Mozambique 

Mexico 

Nicaragua 

Philippines 

Tanzania 

Serbia 


Vietnam 
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Thematic analysis 

Three themes emerged from initially “piling” things that went together into codes 
(Ryan & Bernard, 2003); our subsequent organization of codes into themes based on topics 
that were repeated across interviews (Merriam, 2009; Patton, 2002; Ryan & Bernard, 2003); 
and our later collapsing of nine initial themes into three major themes that were relevant for 
program improvement and programmatic strategic planning purposes (Merriam, 2009; Patton, 
2002; Ryan & Bernard, 2003). 

We present the results of our analysis (i.e., our three major themes) sequentially as 
they relate to the evaluation questions that guided our inquiry (Ulin et ah, 2005). We use 
quotation marks to indicate raw data (i.e., the actual words of our key informants) on which 
we based our analysis and interpretation (Guest & MacQueen, 2007; MacQueen et al., 2007). 
Our first theme provides a clearer picture of the impact that involvement with our program 
had on our key informants and their organizations in low- and middle-resource countries. Our 
second theme relates to the challenges that our course alumni had in establishing and 
maintaining public health management capacity development programs in their countries. 
Our third theme concerns our key informants’ thoughts on new strategies to assist them in 
developing public health management capacity in their countries 

Theme 1 - Impact. 

Individual Impact. 

Our key infonnants shared that their participation in the MIPH course led to individual 
career advancement and professional growth (e.g., improved knowledge, skills and aptitude 
for analytical thinking). Others shared that participating in the course led them to pursue a 
master’s degree in public health. 

One of our key infonnants said she was beginning her career when she participated in 
the MIPH course, and knew little then about public health management. After the course, she 
felt better equipped with improved skills, and her career advanced as a result. She is currently 
a leader in epidemiology within her Ministry of Health. She shared with us how she viewed 
her entire career through the lens of before and after participating in the MIPH course: 

I could divide the important part of my professional life [into] before and after 

[the course]. 

Another key infonnant, who is now a health district director in his country, also said 
that the program prepares participants to advance their careers: 

It lays the foundation to catapult people in some way for higher level things... 

Two other key infonnants, who hold positions within their Ministries of Health, 
reflected on how the knowledge and skills they acquired were useful when they were engaged 
in refonning the public health sector in their countries. One said: 

Now I see the connection with [the] course tools...with [my country’s] existing 

health sector reform...that is, to develop the local government [and] public 

health staff in capacity building. 
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Programmatic and Organizational Impact. 

Our key informants also said that the opportunity to collaborate with the CDC 
(through the process of receiving technical assistance from our program) positively impacted 
their countries’ public health management capacity development programs. It increased the 
Ministry of Health’s awareness that good management is an essential component of effective 
public health programs. One of our key infonnants said: 

...When CDC consultants come here, and meet with different Ministry of 
Health leaders ...and talk about the training program...there is more buy in 
and [it] make[s] our training program better. 

Another key infonnant translated the MIPH course curriculum into her country’s 
native language, and the curriculum was incorporated into her university’s academic public 
health program. She explained that her experience in the course gave her public health 
management skills that she could share with her students. She also said that her students 
would find these skills and tools useful in public health practice, demonstrating that an 
individual participating in the course could have multiple impacts on his/her organization and 
beyond. 

Our key infonnants additionally shared with us that their organizations used the 
management tools that were taught during the MIPH course and that individual and 
organizational performance had improved. One said: 

...I think that [our organization] has been fortified since our capacity of 
analysis has evolved [in] an organized way and with useful tools. All [our] 
project managers [have] been training on issues from [the] course and these 
are used for all of us in our daily work. 

Another key infonnant offered: 

...Most of the lessons [are] still retained...some have been incorporated into 
[our] organization [al] structure.... 

Theme 2 - Factors Impacting Program Development Efforts. 

A manager of an HIV clinic and testing center suggested that it was critical to have 
institutional support if a public health management capacity development program was to be 
developed and successfully maintained. He additionally offered that it was very important to 
get organizational buy-in before initiating any program development efforts, because 
institutional funding and support were essential: 

...[program development] was difficult ...acceptance from the head [i.e., the 
organization’s leader] and budget was very hard. 

Another key infonnant, a senior Ministry of Health official, suggested that a critical 
factor was having a champion within an organization who had time available and the 
responsibility to move a public health management capacity development program forward: 
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[The] secret is to have someone who is highly placed, who has the 
responsibility, doing it specifically... 

Other key informants spoke of the role that technical assistance played in their public 
health management capacity development efforts. One key informant, who holds an 
academic position in a school of public health, noted that her department provides 
management training to health staff throughout her country. She voiced that technical 
assistance from our program was important. 

We really need the technical assistance...that is the follow-up and the 
assistance from the consultants to the [course] participants. 

Theme 3 - New Strategies for Developing Public Health Management Capacity. 

Regional Approach to Public Health Management Training. 

Our key infonnants expressed mixed views about coming to the United States, (i.e., 
Atlanta) to participate in the MIPH course, rather than providing the training regionally in 
different parts of the world. Some said that coming to the U.S. to participate in the course was 
a challenge because it required a substantial financial and time commitment: 

To come to Atlanta ... to get sponsorship is very expensive. It’s like ... 
something like a chance. 

However, some spoke of the positive aspects of coming to the U.S. to participate: 

Those [participating] in Atlanta - [have] better training...better English 
language... [are] more confident...[and] more dedicated to the approach. 

Our key informants’ views often conflicted. At times the same individual expressed 
both challenges and benefits of participating in the course in the United States and in a region 
closer to home. Consequently, no clear conclusion emerged about whether it is better to offer 
the course in the United States or in regions throughout the world. Overall though, our key 
informants did express slightly more interest in our offering the MIPH course in various 
regions outside the United States. One key infonnant from Asia, said: 

It is much more relevant to learn experience in Asian countries than other 
countries like in Africa or in Latin America or something like that. I am just 
thinking-the countries in Asia, we have so much more similar circumstances... 

Another key informant also spoke positively about a regional approach: 

[European] public health problems [are] different from African and Asia. I 
wouldn’t say this was a problem, but a different perspective...we have common 
history with similar problems in health care. 

Distance Learning. 


Our key informants also expressed considerable interest in distance learning. They 
said that they (and other public health professionals in their countries) would engage in 
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distance learning to support their personal career development and to affiliate with an 
international organization. 

One key infonnant, a Ministry of Health official, said that a distance learning course 
would be a good way to teach management skills to public health personnel in his country. 
He also said that the opportunity to receive a certificate of achievement from an international 
organization would encourage public health personnel to complete a distance learning 
program: 

If they get a certificate at the end of the day ...people are interested to link with 

international organizations. 

Concerns about distance learning were also voiced. Several key informants noted that the 
Internet is not always readily accessible or reliable in their countries. One key informant from 
a Ministry of Health spoke of challenges in accessing the Internet from the field: 

If it [the instruction] is informal - no time deadlines, that is possible. People 

are working in the field - so it would be hard - we do not always have access 

[to the internet]... 

Another key infonnant, a district health director, said that because public health 
personnel have many competing priorities at work, it may be difficult to complete an on-line 
public health management training program. While distance learning is promising in theory, 
he expressed that some staff may have a problem in adhering to a set time schedule and 
committing to the hours: 

Distance learning is great. The problem is with adherence... 

Other Programs and Services of Interest 

Our key informants expressed interested in our providing more technical assistance to 
support their public health management capacity development efforts. They specifically 
requested assistance in the areas of monitoring and evaluation, publishing in professional 
journals, decentralizing a national public health system, and advocating for the importance of 
public health management. 

One of our key informants from a Ministry of Health suggested that those who 
received training in CDC’s public health management development course are ready for a 
more advanced course. She suggested a leadership program would be beneficial to help 
advance public health management capacity programs to the next level. She also said that her 
training in our course, and her ability to focus specifically on management related issues, has 
made her an asset to her organization. Others also expressed interest in leadership programs 
and public health management training opportunities tailored to specific levels of a 
professional career (i.e. beginning and mid-levels). 

Discussion 

We conducted our qualitative evaluation to contribute to the improvement of our 
global public health management capacity development program and to produce useful data 
for strategic planning to position our program for additional public health impact. We discuss 
what we gleaned about future programmatic direction, and our approach and lessons learned 
below. 
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Implications for program improvement and strategic planning 

Our inquiry enabled us to learn about factors that may have contributed to or hindered 
public health management capacity development efforts in our key informants’ countries. 
They shared that institutional support (e.g., dedicated funding, sufficient staffing, and a 
commitment from organizational leaders), and having a champion to move a public health 
management capacity development program forward could positively contribute to program 
development efforts. These findings suggest that we could improve the way in which 
applicants are selected for our public health management development programs by 
requesting (a) letters of institutional support indicating that the applicant will have sufficient 
financial and staffing resources to develop a management capacity development program; and 
(b) candidate essays about their time availability and interest in developing public health 
management capacity. 

In terms of new strategies, we learned that our key informants’ were interested in 
improved current services as well as new ones. They were interested in our course being 
offered in alternate sites to Atlanta, GA; training tailored to specific professional levels (i.e., 
beginner, mid and leadership); distance learning, particularly when linked with the ability to 
obtain a certificate; and additional technical assistance. Our recommendations therefore 
included to consider offering our course in other countries and regions of the world; offering 
public health management programs geared to specific career-level; and exploring distance 
learning opportunities, with the recognition that there could be potential challenges related to 
this endeavor (i.e., reliable Internet access in some places in the world, the challenges of 
adhering to online engagements, and that a certificate would be an incentive to complete the 
program). We subsequently shared these recommendations emanating from our findings with 
those directing our program’s strategic planning efforts. 

Sustainability 

Although our evaluation was not designed to learn which factors contributed to MIPH 
alumni successfully developing and sustaining public health management capacity building 
programs in their countries, we learned a little about this through our key infonnant 
interviews. We learned that collaboration with our program and the technical assistance that 
we provided may have contributed to their program development efforts. Future inquires may 
want to therefore explore more fully the relationship between ongoing collaboration with the 
CDC and program sustainability, given that program sustainability without ongoing assistance 
is a desired outcome of CDC’s global public health efforts. 

Limitations 

Our sample size was determined by time and staffing constraints, rather than by 
reaching a point of natural saturation related to emergent themes. Our inquiry could have 
potentially yielded richer results if we had sufficient resources to interview more key 
infonnants from countries representing various levels of program development. We were 
limited though by the practicalities of time and resources (both financial and human). 

Our key informants comprised our most active course alumni in their countries. What 
we learned may have been very different had we interviewed alumni who returned to their 
countries after participating in the course and did not remain active in public health 
management. Our purposeful sampling strategy of selecting key informants from amongst our 
most active alumni may have therefore biased our findings, particularly in regards to 
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individual and organizational impact. Additionally, our efforts to glean infonnation about 
organizational impact could have been strengthened if we had interviewed the supervisors of 
our key informants (in addition to our key informants themselves). Doing so would have 
enabled us to triangulate our data, particularly if we combined what we learned from 
interviewing our key infonnants with what we might have learned from interviewing 
supervisors, and if we had augmented this with organizational case studies. 

We also learned that our inquiry was wide, but possibly not deep. Our inquiry was 
multi-purposed because we wanted to provide data for both program improvement and 
strategic planning purposes. Our questions were therefore broad, and we gleaned a little 
information about a number of different topics. To inquire more deeply into a topic, we 
suggest that an evaluation have a more limited purpose. 

Most of our key informants were not native English speakers. Therefore, we cannot be 
sure that our interpretation of what they said always reflected their intended meaning. In 
addition, one key infonnant was not comfortable being interviewed by telephone in English 
and requested to respond to our questions in writing. Our interpretation of his answers may 
have differed had we interviewed him by telephone. 

Relationship to the literature 

We may not have captured all strategic planning efforts in our literature review 
because organizations may not necessarily be calling these activities strategic planning, per 
se. For example, organizations may have done evaluations, studies, or mapping exercises 
with the intent of strategically contributing to their programmatic efforts, yet they may not 
have ascribed the term “strategic planning” to these particular activities. In addition, many 
more health care and public health entities may be conducting strategic planning than are 
choosing to document these activities in an evidence-based way in the literature. 
Consequently, the prevalence of strategic planning in health care and public health settings 
likely exceeds what we found in our literature review. 

With those caveats noted, we did find an emerging evidence base on using qualitative 
methods in strategic planning, particularly within healthcare settings. We found fewer articles 
providing examples of using qualitative methods in public health strategic planning. Our 
inquiry thus contributes to the emerging evidence-based literature on using qualitative 
methods in strategic planning, and particularly to filling the gap on using qualitative methods 
in public health strategic planning. 

Implications for readers 

We conducted a qualitative program evaluation, and not qualitative research. Our 
findings on program improvement are thus not generalizable to other programs. What may be 
useful to others is that we learned that using a qualitative approach was helpful for both 
program improvement and strategic planning purposes. We found few examples in the 
literature to help guide us, particularly within the public health arena. Our work thus 
contributes to the emerging literature on using qualitative methods, particularly key informant 
interviews, within strategic planning. Readers may therefore want to consider the feasibility 
and utility of including a qualitative evaluation component (including the use of key 
infonnant interviews) as part of their strategic planning efforts. 

We also learned that it was important to reflect the perspective of our program 
beneficiaries in strategic planning. Giving voice to our key infonnants through semi- 
structured interviews allowed us to learn from their experiences and contexts, and therefore 
facilitated a more in-depth understanding of our program. It enabled us to improve our current 
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program offerings and to explore and ultimately implement new strategic directions. Our 
experience was therefore that program beneficiaries were a critically important source of data 
in strategic planning. This is not often highlighted in the literature describing strategic 
planning processes. Readers engaging in strategic planning may therefore want to ensure that 
the context and experiences of their program beneficiaries are given voice to as part of their 
strategic planning process. 

Our review of the literature also led us to conclude that there is a need for more 
evidence-based contributions on how qualitative methods can be used in strategic planning in 
health related fields, particularly in the area of public health. We therefore encourage readers 
engaging in such processes to consider documenting their evidence-based approaches within 
the literature. 

Opportunities for future evaluative work 

We learned that several key informants progressed in their careers after participating 
in our public health management development course. We also learned that some 
incorporated course content and management techniques into their Ministries of Health and 
academic institutions, including in managing public health sector decentralization and reform, 
and that this had positive organizational impact. Our analysis thus provides some rich 
preliminary data that could serve as the basis for exploring our program’s impact at both 
individual and organizational levels in a more in-depth way. We also gathered data on factors 
that may have helped or hindered efforts to develop public health management capacity in our 
key informants’ countries. We could investigate these factors further, in conjunction with 
questions related to what contributes to the sustainability of public health management 
capacity development programs, for purposes of our program’s ongoing improvement. 

Closing thoughts 

Our qualitative evaluation gave us a better understanding of our key informants’ 
experience and perspectives. The themes and recommendations that emerged from our 
qualitative evaluation contributed to program improvement and suggested new directions for 
global public health management programmatic content and service delivery. Our findings 
were thus shared and became one of the inputs informing our program’s strategic planning 
process. Our qualitative evaluation had direct impact on our program’s strategic plan and 
direction. Based on our findings and other inputs, our program is now offering its public 
health management capacity development course in other parts of the globe, a leadership 
course in Atlanta, GA, and increased distance learning offerings. 

The richness of our findings demonstrates that a qualitative evaluation methodology 
can not only be useful for purposes of public health program improvement, but also for 
infonning broader strategic planning purposes. Our inquiry provides a concrete example of 
how qualitative methods, specifically key infonnant interviews, can provide useful data for 
strategic planning within public health settings. It may be useful to readers who are interested 
in conducting strategic planning within public health and other related areas including health 
care, mental and behavioral health, and the social sciences. 
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